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Abstract 

The NASA Langley MDO method evaluation 
study seeks to arrive at a set of guidelines for using 
promising MDO methods by accumulating and an- 
alyzing computational data for such methods. The 
data are collected by conducting a series of re- 
producible experiments. In the first phase of the 
study, three MDO methods were implemented in the 
iSIGHT* framework and used to solve a set of ten rel- 
atively simple problems. In this paper, we comment 
on the general considerations for conducting method 
evaluation studies and report some initial results ob- 
tained to date. In particular, although the results are 
not conclusive because of the small initial test set, 
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Introduction 

Multidisciplinary Design Optimization (MDO) 
problems are optimization problems that describe 
complex coupled engineering systems. The systems 
are composed of physically interacting subsystems de- 
scribed by disciplinary analyses , each of which pos- 
sesses a certain degree of autonomy but depends on 
other subsystems via a number of couplings, also 
known as interdisciplinary variables. 

We distinguish MDO formulations from optimiza- 
tion algorithms in the following way. MDO formula- 
tions belong to an area that studies MDO problem 
definitions, including problem decomposition and in- 
tegration. To analyze an MDO formulation, one con- 
siders such attributes as consistency, equivalence to 


other formulations, optimality conditions, and sen- 
sitivity of solutions to various perturbations. Opti- 
mization algorithms are used to solve a particular 
MDO formulation. It is then appropriate to speak 
of local convergence rates and of global convergence 
properties of an optimization algorithm applied to a 
specific formulation. An analogous distinction exists 
in the field of partial differential equations. On the 
one hand, equations are analyzed in terms of regu- 
larity, well-posedness, and the existence and unique- 
ness of solutions. On the other, one considers numer- 
ous algorithms for solving differential equations. The 
area of MDO methods studies MDO formulations 
combined with optimization algorithms, although at 
times the distinction is blurred. It is important to 


this paper, we focus more on formulations, although 
optimization algorithms play a role as well. 

A sizable and ever growing body of methods and 
their variants has been proposed for solving MDO 
problems. Yet, there is much speculation, but limited 
computational or analytical substantiation of practi- 
cal applicability and algorithmic properties of MDO 
methods. An informative computational method 
study — of different scope and intent — was done by 
Shubin [2]. Haim et al. [3] compare performances 
of several nonlinear programming software packages 
on an MDO problem, given one specific formulation. 
However, in general, a practitioner of MDO has little 
basis for selecting a method among those appearing 
in print. 

Several ongoing efforts at NASA Langley are 
aimed at addressing this deficiency. The present pa- 
per will acquaint the reader with the initial results of 
the first phase of a method evaluation study initiated 
last year. The objectives of the study are as follows: 
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• Accumulate computational data on the perfor- 
mance of promising MDO methods. 

• Compare the practical performance of the 
methods with their proven or conjectured an- 
alytical properties. 

• Arrive at a classification of the methods and 
problems amenable to the methods, as well as 
a set of guidelines for using specific methods. 

• Establish “standards” or guidelines for system- 
atic, easily reproducible, method testing proce- 
dures. 

We consider the last objective exceedingly im- 
portant. Numerical results presented in publications 
about MDO methods are rarely easily reproducible 
by other researchers. This may be due to legitimate 
h th 1 it il bilit f 


(http : // f mad-www . larc . nasa . gov/mdob/MDOB/) .§ 

Interested readers are invited to make contributions 
to this site after familiarizing themselves with the 
submission requirements. 

MDO Methods 

During Phase I of the study, we collected numer- 
ical data on the Multidisciplinary Feasible method 
(MDF), the Collaborative Optimization approach 
(CO), and the Individual Discipline Feasible method 
(IDF). MDF is a mathematical idealization of the 
conventional approach to MDO. The nomenclature 
was introduced by Cramer et al. [5]. MDF was im- 
plemented to serve as a baseline result in this study. 
Antecedents of CO [6] and IDF [5] can be traced to 
work on optimization of large systems, such as that 
of Wismer [7] and Lasdon [8]. Both CO and IDF 
are aimed at solving large problems with a narrow 


validation, and evaluation of methods extremely diffi- 
cult, if not impossible, because one can always argue 
that a particular result is due strictly to an implemen- 
tation and not to an intrinsic property of a method 
under consideration. A remedy is, of course, to ascer- 
tain that the test can be replicated at least for simple 
problems, thus providing a basis for legitimate discus- 
sions of implementations vs. method properties. 

Our objectives present a formidable task, since 
comparing methods intended for solving even the con- 
ventional nonlinear programming problem is notori- 
ously difficult, given the limitations of the problem 
selection, the implementation, and many other vari- 
ables. However, our aim is not to declare one method 
superior to another. Instead, by accumulating com- 
putational data, we seek to understand under what 
circumstances the use of a specific method may be 
advisable. 

In this paper, we comment on testing, in general, 
and describe our testing procedures and their limita- 
tions, in particular. We give detailed results for one 
test problem, followed by a summary of numerical re- 
sults for all problems, as well as conclusions available 
to date. 

A record of all tests and numerical results can be 
found in a forthcoming NASA contractor report [4]. 
A complete record of all tests, codes, and descrip- 
tion files can be found at the NASA Langley Multi- 
disciplinary Optimization Branch (MDOB) method 
evaluation site accessible via the MDOB homepage 


gramming problem of the following form: 

minimize f(x,u(x)) 
subject to h(x,u(x)) = 0 (1) 

g(x,u(x)) > 0, 

where, given a vector of design variables a?, the state 
variables u(x) are defined via a block system of equa- 
tions, 


A(x, u(x)) 


A 1 (x,u 1 (x),...,u n (x)) 
\ A n (x,ui(x), . ,.,u N (x)) 


= 0 , 


where N is the number of blocks. In the context of 
MDO, the blocks of the system usually represent the 
state equations for the disciplinary analyses and the 
necessary interdisciplinary couplings. The equations 
are known as the Multidisciplinary Analysis (MDA) 
system. 


Multidisciplinary Feasible Method 

The MDF formulation is a conventional method 
for solving problem (1). It is an example of the vari- 
able reduction approach to nonlinear programming, 
where only the design variables x are used as inde- 
pendent optimization variables. The statement of the 
problem is unchanged from formulation (1). There- 
fore, theoretically, the convergence properties of any 
optimization algorithm applied to MDF are just its 


§ As the paper is going to press, the NASA Langley web site is being reorganized. If the link is inoperative, please search for 
keywords. 
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At each iteration of the optimization procedure, 
the design variable vector x is input into the MDA 
system. The system is then solved for the state 
vector u, thus reducing the dimension of the opti- 
mization problem. The solution of the MDA system, 
i.e., the multidisciplinary function evaluation, is fre- 
quently performed via the block Gauss-Seidel proce- 
dure (fixed-point iteration). The MDA system can 
be, in principle, solved by any method for solving 
nonlinear equations, such as the Newton’s method. 
One should note, however, that unless a globaliza- 
tion strategy is used, there is no guarantee that a 
method for solving nonlinear equations will solve the 
MDA problem from arbitrary initial points. 

MDF has been in use ever since nonlinear pro- 
gramming techniques were first applied to engineer- 
i ti i ti bl d h i 11 


allel, are nonlinear programming problems whose ob- 
jective is to minimize the discrepancy between the 
shared variables of the subsystems while satisfying 
the disciplinary constraints. 

Braun et al. [9] comment on implementation and 
performance features of CO. A detailed discussion of 
CO’s analytical and computational properties can be 
found in Alexandrov and Lewis [1]. The latter work 
contains complete, precise notation of CO and a sim- 
plified notation useful in initial implementation. We 
use the simplified notation here. 

Assume that for each disciplinary subsystem i, 
given a vector of design variables Xi , the correspond- 
ing vector of responses Ui(xi) is computed via the 
analysis Ai, and the constraints 

9i{xi,u(xi)) > 0, i=l,...,N, 


At each iteration of optimization, multidisciplinary 
equilibrium (or feasibility) is maintained via MDA. 
The term “Multidisciplinary Feasible” refers to this 
property. Given a careful statement of a particular 
problem and a good optimization algorithm applied 
to the formulation, MDF can be efficient. 

The main drawback of MDF is its extreme ex- 
pense. First, the method is costly to implement, be- 
cause a practitioner of MDO has to face the diffi- 
cult problem of analysis integration. Second, a com- 
plete MDA must be done not just at every iteration, 
but also for computing derivatives, if finite-difference 
derivatives are used. Moreover, optimization algo- 
rithms applied to MDF are sensitive to the conver- 
gence of MDA, in general, and the fixed-point itera- 
tion, in particular, which may be detrimental to the 
robustness of the method and its speed. Finally, the 
method is not immune to failing because of attempts 
to process points that cannot be analyzed. 


variables x. Then auxiliary vectors & and are in- 
troduced to represent the shared components of X{ 
and Ui at the system level. 

The system-level optimization problem of the CO 
formulation has the form 


minimize /(£,z/) 
subject to C(£, is) = 0, 


( 2 ) 


where £7(£, v) is the system of N compatibility con- 
straints Ci(&, 2/,*), each one having the form 

Cite, Vi) - ^01 & - X* II 2 + II Vi - Ui{x*i) || 2 ], 

where x* is the solution of the z-th subsystem opti- 
mization problem: 

minimize |[|| & - || 2 + || Vi - Ui(xi ) || 2 ] 

subject to gi(xi,u(xi)) > 0. 


Collaborative Optimization 

In the CO approach, problem (1) is decomposed 
into a number of subsystems, usually along the disci- 
plinary lines. The problem is then reformulated as a 
bilevel programming problem. 

A system objective / is selected for the system 
level. The system-level constraints are the so-called 
“compatibility” or “coupling” constraints designed to 
bring the system into multidisciplinary equilibrium. 
The values and derivatives of the compatibility con- 
straints are computed by solving the lower-level, sub- 


The actual system compatibility constraint can be 
stated as a sum of the individual constraints. Other 
forms of the system-level compatibility constraints 
and subsystem problems exist ([6], [10], [11], [12]), 
but in Phase I, we focused on the CO formulation as 
shown above, as it is the most promising one [1]. 

CO has a number of attractive features. First, it 
dispenses with MDA. Instead of requiring multidis- 
ciplinary feasibility at each iteration of optimization, 
the feasibility is attained in the system optimization 
problem via the introduction of compatibility con- 
straints. Thus, each iteration is feasible with respect 
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to individual analyses, but not multidisciplinary fea- 
sible until a solution is reached. 

CO has the very appealing attribute of disci- 
plinary autonomy with respect to both the analyses 
and optimization, resulting in a relatively easy proce- 
dure for integration of the disciplinary analyses. The 
analyses can be processed in parallel. Another signif- 
icant feature is the maintenance of intradisciplinary 
feasibility at each system iteration, which is impor- 
tant from the application perspective. As MDF, the 
method makes full use of the disciplinary analysis 
software. 

The limitations of CO have to do with attain- 
ing multidisciplinary feasibility and with reformulat- 
ing the original, conventional nonlinear programming 
problem into a nonlinear bilevel programming prob- 
lem. 

In particular, the method is intended for prob- 
lems where the interdisciplinary coupling has small 
bandwidth. Problems with many couplings are not 
expected to benefit from this formulation. The for- 
mulation has more optimization variables than that 
arising from MDF, but given the first limitation, the 
increase in the number of variables should not be 
great, as the number of auxiliary variables depends 
on the bandwidth of the coupling. 

Further, the CO formulation is aimed at solving 
a narrower class of problems than do MDF and IDF ; 
namely, it does not consider general constraints at 
the system level. Technically, general constraints can 
be included at the system level. Their inclusion tends 
to impair the performance of the method, but is un- 
avoidable at times. For instance, in one of our test 
problems, the hub frame design, the objective is to 
minimize the weight of the structure, subject to con- 
straints on the translational and rotational displace- 
ments, stresses in the frame members, and local buck- 
ling of the frame members. The translational and 
rotational displacements of the frame structure are 
global responses and should not be treated as local 
responses at the subsystem level. 

Another limitation is that, despite maintaining 
disciplinary autonomy, CO does not allow explicit 
optimization with respect to disciplinary objectives 
at the disciplinary level. Multiple disciplinary objec- 
tives have to be incorporated at the system level. 

The decomposition procedure may be dictated by 
the application, but it presents a difficult problem for 
cases where structure is not well understood. 

The “small bandwidth of coupling” feature sim- 
ply limits the scope of problems amenable to CO. In 
addition, CO suffers from a difficulty that has to do 
with reformulating a nonlinear programming problem 
into a general, nonlinear, bilevel optimization prob- 


lem. As such, CO is inherently difficult to solve by 
means of software intended for conventional, single- 
level, nonlinear programming problems. While the 
CO formulation is equivalent to the original nonlinear 
programming problem with respect to the solution 
sets, the formulation is not equivalent to the origi- 
nal problem with respect to optimality conditions [ 1 ]. 
This is an example of two problem formulations that 
are equivalent with respect to their solution sets, yet 
are drastically different numerically. 

Based on theoretical analysis of CO [ 1 ], it is ex- 
pected that much fine-tuning would be required to 
implement the method for a specific problem and 
that convergence behavior of conventional optimiza- 
tion methods applied to the CO formulation might 
be erratic. 


Individual Discipline Feasible Method 


The IDF formulation provides another approach 
to “breaking” the expensive MDA iteration. The 
words “Individual Discipline Feasible” refer to main- 
taining disciplinary feasibility at each optimization 
iteration, but not multidisciplinary feasibility until a 
solution is reached. 

Various forms of IDF are described by Cramer et 
al. [ 5 ] and Lewis [ 13 ]. To state the formulation, let 
us assume that the system consists of two subsystems 
and let us write the disciplinary analyses Ai and A2 
in more detail as follows: 


A(x , u(x)) 


A 1 (x,ui(x),u 12 (u 2 (x))) \ _ Q 
A 2 (x,u 2 i(ui(x)),u 2 (x)) ) 


where Uij represent the interdisciplinary flow of in- 
formation from discipline j to discipline i. Then the 
IDF formulation introduces auxiliary variables X12 
and X21, and the optimization problem can be stated 
in the following form: 


minimize f(x , u 1 (x 1 ,x 12 , u 2 {x , £21))) 
subject to X12 — Ui2(u2(x, £21)) = 0 
X21 - U2i(u 1 (x J x 12 )) = 0 
h(x J u 1 (x 1 ,x 1 2,u 2 (x,x 2 i))) = 0 
g(x, u 1 (x lj x 12 , u 2 (x, x 2 i))) > 0 , 


where, u i{x,xi 2 ) and U2(x, X21) are computed by 
solving the disciplinary equations 


A 1 (x,u 1 (x,x 12 ),x 12 ) = 0 

A2{x,X2UU 2 {x,X2l)) = 0 . 


Thus, the IDF formulation is a single-level, nonlinear 
optimization problem. 

On the positive side, IDF is trivially equivalent to 
the original nonlinear programming problem and is 
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thus easy to analyze. That is, the optimality condi- 
tions of the original problem hold for the IDF formu- 
lation. The convergence properties of optimization 
algorithms applied to IDF are those of the algorithms 
applied to conventional nonlinear programming prob- 
lems. Given a good solver for equality constrained 
optimization problems, the method is expected to be 
efficient. 

IDF assures full use of disciplinary analysis soft- 
ware. 

Similarly to CO, IDF is intended for problems 
with small bandwidth of interdisciplinary coupling, 
and the problem of decomposition is similar to that 
of CO. Unlike CO, IDF is not limited to problems 
without general constraints at the system level. 

Also similarly to CO, formulations that arise from 
IDF have more optimization variables that those aris- 
ing from MDF. 

Importantly, although IDF maintains autonomy 
with respect to analyses, it lacks CO’s autonomy with 
respect to disciplinary optimization. That is, while 
the analyses are performed autonomously during the 
analysis stage, the coupling is restored during the op- 
timization step computation. This brings back the 
difficulties of integration. 

Another weakness of IDF is its treatment of the 
disciplinary constraints. They are either ignored in 
descriptions of the formulation or simply relegated to 
the system, despite the need to handle disciplinary 
constraints at the disciplinary level — a wish usually 
expressed by practitioners of MDO. 

Approach 

We selected initial sets of ten problems, three 
methods, and a set of metrics to record during data 
collection. During Phase I, all methods were imple- 
mented in iSIGHT, which is a computational frame- 
work for multidisciplinary design optimization, pro- 
duced by Engineous Software, Inc. The data have 
been compiled and preserved for evaluation. The 
iSIGHT problem description files are available to any- 
one who would like to duplicate the results or improve 
on them. 

There has been much recent work in the area 
of approximations for engineering optimization, and 
Collaborative Optimization has been reported ([14], 
[15]) to show improvement due to the use of response 
surface methodology. In this study, we have not at- 
tempted to use approximations. 

Implementation and Its Limitations 

In this study, we are concerned with evaluating 


MDO methods, which is a notion composed of “for- 
mulations” and “algorithms”. A formulation is a 
statement of the problem to be solved by an opti- 
mization algorithm. An algorithm is then a set of 
steps performed to solve a formulation. The two are 
very much interrelated, and are even difficult to dis- 
tinguish for some methods. MDF, CO, and IDF are 
all formulations, but they each may be implemented 
in a variety of ways and their evaluation will be af- 
fected by the use of specific optimization algorithms. 
This makes the evaluation extremely difficult. How- 
ever, trends in performance can still be discerned. 

The implementation of the methods and the prob- 
lems, as well as the software used to solve the prob- 
lems, have a direct bearing on the performance of the 
methods. Moreover, we understand completely that 
to use a particular method to its full advantage, es- 
pecially a method for solving problems as difficult as 
those that arise in MDO, one will, by necessity, per- 
form a significant amount of fine-tuning, both in the 
problem statement and in all areas of implementa- 
tion. 

Taken to its extreme, however, this principle 
means that, given any problem and any initial 
method, with a sufficient amount of time the method 
can be “fine-tuned” so that the problem can be solved 
with reasonable efficiency. The danger of such an 
extreme is that what practically amounts to a new 
method has to be developed for each problem. The 
questions are then whether an original method can 
still be discerned after such an exercise and just how 
much of fine-tuning is required to produce reasonable 
performance. 

An important consideration in making decisions 
about fine-tuning is the available resources for solv- 
ing a specific problem. If an expensive problem will 
be solved many times, it makes sense to fine-tune the 
method and the problem to assure optimal perfor- 
mance. However, in some projects, it is anticipated 
that the problem under consideration will be solved 
only a few times due to its extreme expense. In such 
cases, one cannot afford fine-tuning and one has to be 
reasonably certain of a method’s robustness before its 
application. 

With this in mind, we made a decision to avoid 
fine-tuning as much as possible in the first phase of 
the study, in order to obtain an idea of what a reason- 
ably intelligent user, who is not a method developer, 
will face when implementing a specific method. In 
further studies, a higher degree of fine-tuning the im- 
plementation is planned. 

The choice of optimization software for solving the 
formulated problems and subproblems was limited by 
the set available in iSIGHT. 
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Problem 

Title 

Source 

1 

Conceptual Ship Design 

[14] 

2 

Electronic Packaging 

[16, 17] 

3 

Power Converter 

[18, 16] 

4 

Speed Reducer 

[19, 16] 

5 

Combustion of Propane 

[20, 16] 




10 

Three-Component Separation 

[22] 


Table 1: Test problem set for Phase I. 


Problem 

Method 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

MDF 

# Variables 

6 

8 

6 

7 

4 

4 

120 

80 

48 

52 

# Constraints 

7 

3 

4 

11 

4 

4 

764 

115 

38 

40 

CO 

System: 

# Variables 

11 

5 

6 

2 

4 

8 

40 




# Constraints 

5 

2 

6 

3 

2 

6 

6 

- 

- 

- 

# Subsystems 

5 

2 

2 

3 

2 

2 

2 

- 

- 

— 

Total # subsystem variables 

18 

12 

12 

11 

7 

12 

120 




IDF 

# Variables 

14 

12 

12 


6 

8 





# Constraints 

11 

5 

6 

- 

6 

6 

- 

- 

- 

- 


Table 2: Problem dimensions. 


Within that set, the choice was somewhat subjec- 
tive and dependent on experience. At this stage, we 
have not conducted conclusive studies on the sensi- 
tivity of a particular formulation to the choice of the 
optimizer. 

All three methods were implemented in the 
iSIGHT framework, using MDOL, the iSIGHT MDO 
Language. The implementation was both eased and 
made more difficult because the methods were tested 
within the iSIGHT framework. On the one hand, 
the framework provided a unified approach to testing. 
On the other hand, iSIGHT is designed for handling 
large, “production” problems, and there was signif- 
icant overhead connected with coding smaller prob- 
lems in the early stages of the study. In addition, 
we were restricted to using the optimization software 
available in iSIGHT for solving the optimization sub- 
problems. Overall, we feel that at this initial stage, 
the benefits of using iSIGHT outweighed the disad- 
vantages of the additional work necessary to incorpo- 


rate the problems and the methods into the frame- 
work. 

Implementation within iSIGHT also affected the 
performance of the methods on some of the problems, 
in that stand-alone implementation has resulted in a 
much faster convergence. Future method evaluation 
studies are planned both in and out of iSIGHT. 

Given all the limitations, however, our results are 
available for examination, as is all the software that 
produced the results. Should the authors of a partic- 
ular method or other researchers find that a certain 
trend we discerned has to do strictly with our imple- 
mentation and not with the innate properties of that 
method, we welcome suggestions on how to make the 
tests more informative. 

Finally, promising methods evolve continually, 
their implementation improves with increased under- 
standing of their workings, and the difficulties appar- 
ent at one stage of their development may disappear 
at a more mature stage. 
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Problems 

The ten problems selected for Phase I of the study 
are listed in Table 1. Earlier references exist for most 
of these problems. We indicate the sources we used. 
The reference for the MDOB Test Suite [16] is indi- 
cated for all problems directly downloaded from that 
set, but the primary reference for the problem is the 
other reference. 

At this stage, the problems are not representative 
of realistic MDO applications. However, to actually 
solve a realistic MDO problem may require months or 
even years, which would somewhat incapacitate the 
effort. Therefore, we chose several tractable prob- 
lems as an initial set we could easily handle. Some 
of the problems, such as the Conceptual Ship Design 
Problem, the Electronic Packaging Problem, and the 
Hub Frame Problem, exhibit some of the salient fea- 
tures of realistic MDO problems, albeit not their size, 
complexity, or expense. 

While a method cannot be judged applicable to 
a realistic problem just because it performs well on 
a toy problem, if it does not perform well on rela- 
tively simple problems, that is an indication that the 
method must be further studied before attempting to 
use it for solving realistic problems. 

During future studies, we plan to obtain numeri- 
cal data on solving more realistic problems with the 
methods that are deemed promising after this initial 
testing. A summary of the problem dimensions is 
given in Table 2. In cases where the problems were 
found inappropriate for being solved by a specific 
method, the table contains — •”. 

Note that the constraints of the system-level prob- 
lem for CO are just the compatibility constraints. 

Metrics 

We have considered a number of metrics in eval- 
uating the performance of the methods. Some of the 
metrics are objective and quantitative, the others, 
more subjective and qualitative. Discussion of some 
of the metrics follows. 

Generality. This is a measure of applicability of 
a method to various classes of problems. Our prob- 
lem pool is small at this stage, but as it grows, we 
will classify the problems by their size in terms of vari- 
ables and constraints, smoothness, nonlinearity, com- 
plexity, strength and bandwidth of coupling, presence 
of several objectives, and other features. 

Robustness. Given that each method is designed 
for solving a specific class of problems, this metric 
evaluates the capacity of the method to provide a so- 
lution or a “satisfactory” point for that class of prob- 


lems for a variety of perturbed data. For the initial 
set of problems that we considered, we measured ro- 
bustness by attempting solution from several starting 
points. 

Another, somewhat subjective, but important 
measure of robustness we considered was the amount 
of fine-tuning required to make a method produce an 
answer. For instance, we considered restructuring the 
problems in a number of ways, such as the incorpora- 
tion of slack or squared slack variables, or changing 
the tolerances on constraints and optimality. 

Performance. Performance is a very complex is- 
sue, strongly related to implementation, problem for- 
mulation, and the nature of the problem. In Phase 
I of the study, we quantified performance as work 
performed by each method during every optimization 
procedure. 

The recorded work consists of the total number of 
analyses, with each “disciplinary” analysis counted 
as one function evaluation. Function evaluations 
computed during finite-difference derivative compu- 
tations are included. In particular, for MDF, we 
report the total number of analyses, including the 
average number of analyses taken during fixed-point 
iterations in order to compute MDA, and those neces- 
sary to compute the finite-difference derivatives. For 
CO, we report the number of iterations taken by the 
system-level optimization problem, as well as the sum 
of the function evaluations in each subsystem, includ- 
ing those required for finite-difference computations. 
For IDF, we report the total number of function 
evaluations, including those taken for finite-difference 
computations, times the number of the subsystems. 

Although the dimensions of the design space dif- 
fer for MDF, CO, and IDF, by reporting the total 
number of analyses, we provide as complete a mea- 
sure of work as possible, given the serious limitations 
imposed by a specific implementation of the methods 
and the problem formulations. 

Example: Electronic Packaging 

The Electronic Packaging problem ([16], [17]) con- 
siders a circuit that comprises two resistors mounted 
on a heat sink, resulting in two coupled subsystems: 
electrical and thermal. Operating temperatures af- 
fect component resistance, while resistance, in turn, 
influences the temperature. The objective function is 
watt density. The constraints require the operating 
temperatures of resistors to be below a threshold tem- 
perature and the currents through the two resistors 
to be equal. The problem has eight design variables, 
thirteen state variables, two parameters ( voltage and 



{xi,X 2 ,X 3 ,X A ) 


(£ 5 , Xq, £ 7 , X 8 ) 


Resistance: (tq, U 3 ) 

Electrical SS Thermal SS 


Figure 1: Input-output diagram for 

T°), and may be stated as follows: 

maximize u\ 
subject to /ii = 2/4 — 2/5 

Qi — «n — 85.0 < 0 
#2 = «i 2 85.0 < 0 

0.050 < x\ < 0.15 
0.050 < ^2 < 0.15 
0.010 < ^3 < 0.10 
0.005 < x 4 < 0.05 

10.00 < < 1000.0 

0.004 < < 0.009 

10.00 < ^7 < 1000.0 

0.004 < < 0.009, 

where the first constraint represents the branch cur- 
rent equality, while the next two inequalities are the 
component reliability constraints. 

The eight design variables are as follows: x\ and 
X 2 are the heat sink width (m) and length (m); X 3 
and X 4 are the fin length (m) and width (m): x$ and 
x 7 are the nominal resistance (O) of components 1 
and 2, respectively, at temperature T° ; xq and x$ are 
the temperature coefficients (°K~ 1 ) of the electrical 
resistance components 1 and 2 . 

The state variables are as follows: iq is the watt 
density (watt/m 3 ); iq and U 3 are the resistances (fi) 
at temperatures T\ and T 2 ; U 4 and iq are currents 
(amps) in resistors 1 and 2 ; uq and u 7 represent 
power dissipation (watts) in resistors 1 and 2 ; ug, 
uq 7 and uio are the total circuit current, resistance, 
and power, respectively; uu and U 12 are the compo- 
nent temperatures, T\ and T 2 (°C), of resistors 1 and 
2 ; and finally, tq 3 is the heat sink volume (m 3 ). 

The states are described by the following equa- 
tions: 

Ml = -«lo/«13 


the electronics packaging problem. 


U2 

= 

£5(1.0 + £ 6 (wil - r°)) 

U 3 

= 

£r(1.0+£ 8 («12-r°)) 

U4 

= 

U 3 U 8 /(u 2 + U 3 ) 

U5 

= 

U 2 U 8 /(U2 + U 3 ) 
„,2 ni 

Uq 

_ 

U4U2 

n ,2 n . 

U 7 

— 

U5U3 

U 8 

= 

voltage/ U q 

Uq 

= 

(l.O/tia + l.O/us)" 1 

uio 

— 

U 8 Uq 

Ull 

= 

function^, X2, X3 , X4, iq, u 7 

U12 

= 

function^, aq, £3, £4, uq, u 7 

Ul 3 

= 

X1X2X3, 

where the state 

s uu and U12 are rather invo. 


pressions. The code for this problem can be found 
at the MDO Test Suite web site accessible from the 
MDOB Homepage. 

The reported results are for cases initiated from 
the same starting points, for all methods. 

Fig. 1 depicts interdisciplinary interactions in the 
problem. 

The following three subsections deal with solving 
the problem by using the three methods under study. 

The following conventions are used in all data ta- 
bles. 

• “Convergence” means the attainment of a point 
that satisfies the Karush-Kuhn- Tucker (first- 
order necessary) conditions for a critical point 
of the problem within the maximum allowable 
number of iterations (set at 10,000). However, 
when a particular problem was solved, known 
solutions were always found rather than just 
critical points of the problem. 


8 

American Institute of Aeronautics and Astronautics 



• “Work” is defined as the total number of disci- 
plinary analyses performed. For MDF, work is 
equal to the number of calls to the function eval- 
uation procedure times the average number of 
fixed point iterations per MDA times the num- 
ber of “disciplines”. For CO, the total number 
of function evaluations at the subsystem level 
and the number of system iterations is reported. 
For IDF, we report the number of disciplinary 
function evaluations times the number of “dis- 
ciplines” . 

Function evaluations done during finite- 
difference computations are included in all 
work. 

• The superscript “F” added to the value of the 
final objective indicates failure to converge to 
a critical point within the allowable number of 
iterations. 

• In all tables, “-” means that the problem was 
found inappropriate for a particular method, 
and the tests were not run. 

MDF Implement at ion 

The MDF formulation has eight design variables, 
three explicit constraints, and its statement is that 
of the original problem. The problem was solved 
using the method of feasible directions in iSIGHT. 
The derivatives were computed using finite differ- 
ences with the step size of 1%. The termination cri- 
teria included the satisfaction of the first-order neces- 
sary conditions for optimality, tolerances for the ab- 
solute and relative change in the objective function 
during several successive iterations, and the maxi- 
mum number of allowable iterations. The MDF re- 
sults are summarized in Table 3. 

CO Implementation 

For the CO approach, the system-level optimiza- 
tion problem is given by 

maximize Ci 
subject to ci < 0.0001 
c 2 < 0.0001. 

The system-level problem has five design variables: 
Ci, £2, £3, £11, and £ 12 . 

The Thermal Subsystem optimization problem is: 

minimize ci 
subject to hi = 0.0 

gi = un — 85.0 < 0.0 
g 2 — U 12 — 85.0 < 0.0, 


where 

a — («n — £ll ) 2 + (ui 2 — C12) 2 + («2 — C2) 2 

+ («3-6) 2 + («l -£l) 2 , 

where the variables of the Thermal Subsystem are 
Xi, i — 1, . . . , 4, U 2 , and U3. 

The Electrical Subsystem optimization problem 
is: 

minimize c 2 

subject to gi = uu — 85.0 < 0.0 
g 2 = U 12 — 85.0 < 0.0, 

where 

C2 = (^2— ^2) 2 +(^3— ^3) 2 + (^ll— ^ll) 2 T(wi2— £12)", 

where the variables of the Electrical Subsystems are 
Xi, i — 5, . . . , 8, itn, and «i 2 . 

The system- level optimization problem was solved 
by a combination of the exterior penalty function 
method and the method of feasible directions avail- 
able in iSIGHT. The subproblems were solved us- 
ing a sequential quadratic programming algorithm 
DONLP-implementedin iSIGHT. The CO results are 
summarized in Table 4. 

IDF Implementation 

The compatibility constraints of the IDF formu- 
lation were implemented as inequalities, since that 
seemed to produce better results, given the optimiza- 
tion software available in iSIGHT. The optimization 
problem is: 

maximize u\ 
subject to ci < 0.0001 
c 2 < 0.0001 
hi = U 4 — u 5 = 0.0 

g 1 = un — 85.0 < 0.0 
g 2 = Ui 2 — 85.0 < 0.0, 

where the twelve design variables are Xi , i — 1 , . . . , 8, 
and four coupling variables £ 2 , £3, C11, and £12. 

The Thermal Subsystem evaluates u\, hi, and 

Cl — («H — £ll) 2 + {Ui2 ~ Cl2) 2 , 

while the Electrical Subsystem evaluates 

c 2 = (u 2 — C2) 2 + (^3 — ^3) 2 - 

The optimization problem was solved by a com- 
bination of the exterior penalty function method and 
the method of feasible directions available in iSIGHT. 
The IDF results are summarized in Table 5. 
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1 

7.7944D+01 

2.1663D-08 

6.3972D+05 

1.2188D-03 

83 x 3 x 2 = 498 

2 

6.8363D+03 

-2.8956D-01 

6.3972D+05 

1.2188D-03 

44 x 3 x 2 = 264 

3 

1.5111D+03 

-4.2924D-02 

6.3654D+05 

1.4514D-03 

44 x 3 x 2 = 264 

4 

1.4607D+03 

-1.0249D-03 

6.3694D+05 

1.4211D-03 

35 x 3 x 2 = 210 


Table 3: Electronic packaging: MDF results. 


Initial 

Point 

Initial 

Objective 

(System) 

Initial 

Max. Con. Viol. 

Final 

Objective 

(System) 

Final 

Max. Con. Viol. 

Work 

1 

7.7944D+01 

0.00D+00 

3.51968D+05 

1.00D-04 (ci) 
1.02D-08 (c 2 ) 

110 system iter 
4886 + 8899 = 13785 

2 

6.8300D+03 

-2.89D-01 

6.57163D+05 

2.30D-04 (d) 
1.30D-04 (c 2 ) 

123 system iter 
6315 + 13577 = 19872 

3 

1.5111D+03 

-4.20D-02 

6.50+04^ 

7.60D-03 (ci) 
6.57D-09 (c 2 ) 

138 system iter 
13414 + 12650 = 26064 

4 

1.4607D+03 

-1.00D-03 

6.5D+04 F 

4.80D-03 (ci) 
1.11D-08 (c 2 ) 

94 system iter 
10205 + 9396 = 19701 


Table 4: Electronic packaging: CO results. 


Initial 

Point 

Initial 

Objective 

Initial 

Max. Con. Viol. 

Final 

Objective 

Final 

Max. Con. Viol. 

Work 

1 

7.7944D+01 

2.248D-03 

6.8131D+05 

6.00D-04 (ci) 
1.77D-06 (c 2 ) 

135 x 2 = 270 

2 

6.8363D+03 

-2.890D-01 

6.5367D+05 

1.10D-04 (ci) 
1.00D-04 (c 2 ) 

4488 x 2 = 8976 

3 

1.5111D+03 

-4.200D-02 

6.7740D+05 

6.00D-04 (V) 
1.00D-04 (c 2 ) 

2053 x 2 = 2106 

4 

1.4608D+03 

-1.000D-03 

6.7577D+05 

1.70D-04 (ci) 
1.05D-05 (c 2 ) 

3437 x 2 = 6874 


Table 5: Electronic packaging: IDF results. 


Summary of Results 

The initial set of test problems was solved using 
the iSIGHT MDOL language based implementation 
of MDF, CO, and IDF from several starting points. 
The ratio of the successful runs to the number of at- 
tempted runs is summarized in Table 6. By a “suc- 
cessful run” we mean one that attains a critical point 
of the problem, to a specified degree of tolerance, 
within the allowable maximum number of iterations. 

All runs were originally planned to be attempted 
from twelve starting points. Due to time limitations, 
all the data could not be generated to date. This 


explains the difference in the numbers of attempted 
runs among the problems and methods shown in Ta- 
ble 6. 

During the problem formulation, we discovered 
that some of the problems did not have a suitable 
structure for the CO approach or for the IDF ap- 
proach. We did not discard these problems, as they 
may prove useful for testing other methods. 

Given the small problem sample, and the limi- 
tations mentioned earlier, we cannot make definitive 
statements about the methods under study. However, 
even this sample has demonstrated that the method 
properties predicted by analysis tend to hold. 
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Problem 

Method 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

MDF 

12/12 

12/12 

12/12 

12/12 

3/3 

5/12 

10/10 

1/6 

5/10 

5/6 

CO 

4/4 

2/5 

3/5 

4/5 

5/5 

4/5 

0/5 

- 

- 

- 

IDF 

4/4 

5/5 

4/5 

- 

4/5 

3/5 

- 

- 

- 

- 


Table 6: Summary: number of successful runs / number of attempted runs. 


Problem 

Method 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

MDF 

610 

220 

610 

81 

3234 

5024 

8730 

245 

1574 

1353 

CO 

15626 

19872 

1785 

2102 

837 

40125 

691058 

- 

- 

- 

IDF 

9530 

8976 

382 

- 

544 

932 

- 

- 

- 

- 


Table 7: Summary: representative number of analyses for convergence. 


Problem 

Method 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

MDF 

667 

275 

1025 

77 

10626 

3035 

6887 

245 

1547 

1353 

CO 

13065 

18005 

2983 

2102* 

837* 

40125* 

691058* 

- 

- 

- 

IDF 

9640 

6019 

406 

- 

694 

1071 

- 

- 

- 

- 


Table 8: Summary: average number of analyses for convergence. Note: numbers marked by an asterisk are 
based on a single data point. 


In particular, the following trends were observed with 
respect to convergence and the number of function 
evaluations. 

MDF 

The method converged to known solutions for the 
majority of cases. However, for two of the problems, 
the use of the fixed-point iteration to attain MDA 
caused very unstable behavior, and the optimizer was 
unable to converge from many starting points. 

Given the small sample, the number of function 
evaluations is not conclusive. However, for the cur- 
rent dataset, MDF consistently requires fewer func- 
tion evaluations than does CO (with the exception 
of one problem). MDF did better than IDF for two 
problems, and worse than IDF for three problems. 
Interestingly, IDF did worse on problems that 1 and 
2, that exhibit more typical MDO structure than do 
problems 5 and 6, on which MDF performed poorly. 

CO 

The method was used to solve seven of the ten 


problems. The system-level compatibility constraints 
were actually implemented not as equalities, but as 
inequalities, as this alleviates some of the numerical 
difficulties associated with the equality constrained 
system-level problem ([1]). Some adjustment of the 
tolerances on the system-level constraints and on the 
lower-level convergence criteria had to be done. A 
general qualitative trend can be stated as follows: 
larger tolerances on the system-level constraints and 
smaller tolerances on the lower-level convergence cri- 
teria lead to better chances for attaining overall con- 
vergence. 

Compared to the other two methods, CO typi- 
cally required a significantly larger number of func- 
tion evaluations. It was more efficient than MDF 
only for problem 5, and it was always less efficient 
than IDF. 

CO also seemed to be less robust than the other 
methods, failing to find a critical point more fre- 
quently. On the one hand, this feature confirms the 
analytical properties of CO. On the other hand, this 
may be explained by our formulation of the prob- 
lems, and by not spending a large amount of time on 
fine-tuning the method, which was intentional, as we 
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mentioned. 

Again, for a large test set, with a more evident 
MDO structure, the difference in performance may 
be less pronounced. 

IDF 

Similarly to CO, the compatibility constraints for 
IDF were relaxed to inequalities. 

IDF performed consistently better than CO on 
our small sample of problems. However, one must 
remember that IDF is less attractive than CO with 
respect to the ease of integration and its handling of 
the disciplinary constraints. 

A representative number of analyses required for 
each of the methods, for our test set, is summarized 
in Table 7, while the average number of analyses is 
given in Table 8. 

General Comments 

During the testing, we also considered the less 
tangible metrics, such as the ease of implementation. 
Here, the MDF was not typical because of the na- 
ture of our problems. It was the easiest to implement 
because our problems are small and relatively sim- 
ple. However, in real applications, the integration of 
MDA can be a formidable task. 

For our small test set, CO and IDF were about 
equally easy to implement. For large, more realistic 
applications, we expect that CO will be easier to im- 
plement, because of the way it handles disciplinary 
constraints. 

The current and future method studies are pro- 
ceeding on several fronts. Information is accumulated 
on other methods, such as the multilevel methods 
(MAESTRO [23], [24] [25]) and concurrent subspace 
optimization methods (CSSO [26], [27]); testing pro- 
cedures are being fine-tuned for the methods evalu- 
ated in Phase I of the study; larger, more realistic 
problems are being added to the problem test set. 

In conclusion, while the results are by no means 
complete, we have found this systematic study of 
methods very informative. The tests in general con- 
firmed the expected numerical properties; however, a 
significant amount of further testing would be nec- 
essary to explain the numerical behavior of methods 
definitively. We invite other researchers to contribute 
to accumulating systematic data, that will eventually 
lead to a set of practical guidelines for the use of MDO 
methods (please see the method evaluation web site 
accessible from the MDOB Homepage). 
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